feat(ui-automation): Add rs/1 runtime automation parity#416
Open
cameroncooke wants to merge 66 commits into
Open
feat(ui-automation): Add rs/1 runtime automation parity#416cameroncooke wants to merge 66 commits into
cameroncooke wants to merge 66 commits into
Conversation
commit: |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Validation artifacts with local paths committed to repo
- Removed entire out.nosync/ directory from git tracking and added it to .gitignore to prevent future commits.
- ✅ Fixed: Plan document embeds developer-local filesystem paths
- Replaced all machine-specific absolute paths with portable references or removed them entirely from the plan document.
Or push these changes by commenting:
@cursor push 07a7505ffa
Preview (07a7505ffa)
diff --git a/.gitignore b/.gitignore
--- a/.gitignore
+++ b/.gitignore
@@ -116,3 +116,6 @@
.derivedData
/.pr-learning
/repros
+
+# Validation artifacts
+out.nosync/
diff --git a/docs/plans/ui-automation-agent-optimization-2026-05-13.md b/docs/plans/ui-automation-agent-optimization-2026-05-13.md
--- a/docs/plans/ui-automation-agent-optimization-2026-05-13.md
+++ b/docs/plans/ui-automation-agent-optimization-2026-05-13.md
@@ -12,7 +12,7 @@
- `wait_for_ui` polls snapshots and records the latest usable runtime snapshot while waiting (`src/mcp/tools/ui-automation/wait_for_ui.ts`).
- Prior art: `04b055210b111c8a2d70bdf5c19ca4c6b0d2a479` added RS/1 runtime automation parity, including batch execution, wait predicates, runtime refs, and screen-hash unchanged responses.
-Transcript evidence from `/Users/cameroncooke/Desktop/out.nosync`:
+Transcript evidence from validation run:
- Settings toggles were handled as repeated `tap -> snapshot_ui -> tap` loops (`0058` through `0087`) instead of one snapshot plus one batch.
- Batch syntax was guessed and failed (`tap e7`, `tap --element-ref e7`, `tap-target e21`) before the agent fell back to coordinates (`0281` through `0286`, `0312` through `0327`).
- Scroll and sheet expansion produced repeated gesture/swipe/screenshot/snapshot loops (`0110` through `0144`).
@@ -83,9 +83,9 @@
Use Claude Code, not Codex, for validation runs. Configure Claude Code to use the local source-built XcodeBuildMCP server, then repeat the Weather-app task from `0001_user_message.md`.
Reference commands and artifacts:
-- Existing parser: `/Volumes/Developer/parse_claude_conversation.py`
-- Example source conversation: `/Users/cameroncooke/.claude/projects/-Volumes-Developer-XcodeBuildMCP-example-projects-Weather/b3d6cae2-e274-4a72-92ea-25eaf4f6fcff.jsonl`
-- Export command example: `python3 /Volumes/Developer/parse_claude_conversation.py <claude-jsonl>`
+- Existing parser: `parse_claude_conversation.py`
+- Example source conversation: `.claude/projects/<project-path>/<session-id>.jsonl`
+- Export command example: `python3 parse_claude_conversation.py <claude-jsonl>`
Acceptance signals:
- The settings-toggle sequence is reduced to one `snapshot_ui`, one structured `batch`, and at most one verification call.
@@ -106,11 +106,11 @@
## Validation Results
-Validation artifacts were written under `/Volumes/Developer/XcodeBuildMCP/out.nosync/validation-ui-automation-20260513-215522` with timestamped names. Prior Claude transcripts/exports were not deleted or overwritten.
+Validation artifacts were written under `out.nosync/validation-ui-automation-20260513-215522` with timestamped names. Prior Claude transcripts/exports were not deleted or overwritten.
### Automated checks
-All checks below were run with `XCODEBUILDMCP_AXE_SOURCE_PATH=/Volumes/Developer/AXe`:
+All checks below were run with `XCODEBUILDMCP_AXE_SOURCE_PATH` pointing to the AXe source build:
- Focused UI automation/config/factory tests: `npx vitest run src/mcp/tools/ui-automation/__tests__/batch.test.ts src/mcp/tools/ui-automation/__tests__/runtime-snapshot.test.ts src/mcp/tools/ui-automation/__tests__/snapshot_ui.test.ts src/mcp/tools/ui-automation/__tests__/tap.test.ts src/mcp/tools/ui-automation/__tests__/wait_for_ui.test.ts src/utils/__tests__/axe-helpers.test.ts src/utils/__tests__/config-store.test.ts src/utils/__tests__/project-config.test.ts src/utils/__tests__/session-aware-tool-factory.test.ts src/utils/responses/__tests__/next-steps-renderer.test.ts` passed: 10 files, 192 tests (`focused-vitest-20260513-215522.log`).
- `npm run lint` passed (`lint-20260513-215539.log`).
@@ -118,7 +118,7 @@
- `npm run typecheck` passed (`typecheck-20260513-215539.log`).
- `npm run build` passed (`build-20260513-215553.log`).
- `npm run test` passed: 186 files, 2049 tests (`test-20260513-215553.log`).
-- Post-review fixes were applied for switch batch delay handling and selector-scoped `gone` text waits. Final checks after those fixes passed: `npm run lint`, `npm run format:check`, `npm run typecheck`, `npm run build`, and `npm run test` (186 files, 2052 tests). The final full-test output was preserved at `/tmp/xcodebuildmcp-final-npm-test-20260513.txt`.
+- Post-review fixes were applied for switch batch delay handling and selector-scoped `gone` text waits. Final checks after those fixes passed: `npm run lint`, `npm run format:check`, `npm run typecheck`, `npm run build`, and `npm run test` (186 files, 2052 tests).
### AXe source-build proof
@@ -127,18 +127,18 @@
```json
{
"resolved": {
- "path": "/Volumes/Developer/AXe/.build/release/axe",
+ "path": "<axe-source-path>/.build/release/axe",
"source": "source"
},
"bundledEnvironment": {}
}-The resolved binary reported version staging-main-31-510d4df-dirty, proving the validation used the /Volumes/Developer/AXe source build rather than bundled or PATH fallback.
+The resolved binary reported version staging-main-31-510d4df-dirty, proving the validation used a local AXe source build rather than bundled or PATH fallback.
Claude Code E2E
-Claude Code ran the full original 18-step Weather/Safari task from /Users/cameroncooke/Desktop/out.nosync/0001_user_message.md against the local source-built XcodeBuildMCP server:
+Claude Code ran the full original 18-step Weather/Safari task against the local source-built XcodeBuildMCP server:
- MCP config:
claude-mcp-config-20260513-215817.json - Prompt:
claude-weather-safari-prompt-20260513-215817.md
@@ -153,9 +153,9 @@
{
"command": "node",
- "args": ["/Volumes/Developer/XcodeBuildMCP/build/cli.js", "mcp"],
+ "args": ["<xcodebuildmcp-path>/build/cli.js", "mcp"],
"env": {
- "XCODEBUILDMCP_AXE_SOURCE_PATH": "/Volumes/Developer/AXe",
+ "XCODEBUILDMCP_AXE_SOURCE_PATH": "<axe-source-path>",
"XCODEBUILDMCP_SENTRY_DISABLED": "1"
}
}
@@ -181,8 +181,5 @@
- Safari WebView contents, cookie/sign-in UI, and BBC in-page links were not exposed as tappable RS/1 targets, so Claude used the URL bar for Sport, Premier League, tables, and Brighton. This reached the equivalent end state but was not a real row/link click.
## References
-- Transcript folder: `/Users/cameroncooke/Desktop/out.nosync`
-- Validation artifact folder: `/Volumes/Developer/XcodeBuildMCP/out.nosync/validation-ui-automation-20260513-215522`
-- Parser: `/Volumes/Developer/parse_claude_conversation.py`
-- AXe workspace: `/Volumes/Developer/AXe`
+- Validation artifact folder: `out.nosync/validation-ui-automation-20260513-215522`
- Prior RS/1 commit: `04b055210b111c8a2d70bdf5c19ca4c6b0d2a479`
diff --git a/out.nosync/validation-ui-automation-20260513-215522/churn-analysis-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-221250.json b/out.nosync/validation-ui-automation-20260513-215522/churn-analysis-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-221250.json
deleted file mode 100644
--- a/out.nosync/validation-ui-automation-20260513-215522/churn-analysis-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-221250.json
+++ /dev/null
@@ -1,1008 +1,0 @@
-{
- "jsonl": "/Volumes/Developer/XcodeBuildMCP/out.nosync/validation-ui-automation-20260513-215522/claude-session-94c0a294-37b0-453f-9ac6-774095a4ace0-20260513-215817.jsonl",
- "total_lines": 425,
- "total_tool_calls": 138,
- "all_tool_counts": {
- "ToolSearch": 3,
- "mcp__xcodebuildmcp-dev__session_show_defaults": 1,
- "mcp__xcodebuildmcp-dev__build_run_sim": 2,
- "mcp__xcodebuildmcp-dev__snapshot_ui": 21,
- "mcp__xcodebuildmcp-dev__wait_for_ui": 23,
- "mcp__xcodebuildmcp-dev__screenshot": 19,
- "Bash": 2,
- "TodoWrite": 14,
- "mcp__xcodebuildmcp-dev__stop_app_sim": 1,
- "mcp__xcodebuildmcp-dev__launch_app_sim": 1,
- "mcp__xcodebuildmcp-dev__swipe": 15,
- "mcp__xcodebuildmcp-dev__tap": 16,
- "mcp__xcodebuildmcp-dev__batch": 1,
- "mcp__xcodebuildmcp-dev__gesture": 2,
- "Grep": 3,
- "Read": 2,
- "mcp__xcodebuildmcp-dev__type_text": 6,
- "mcp__xcodebuildmcp-dev__button": 1,
- "mcp__xcodebuildmcp-dev__key_press": 5
- },
- "xcodebuildmcp_tool_counts": {
- "session_show_defaults": 1,
- "build_run_sim": 2,
- "snapshot_ui": 21,
- "wait_for_ui": 23,
- "screenshot": 19,
- "stop_app_sim": 1,
- "launch_app_sim": 1,
- "swipe": 15,
- "tap": 16,
- "batch": 1,
- "gesture": 2,
- "type_text": 6,
- "button": 1,
- "key_press": 5
- },
- "batch_calls": [
- {
- "ordinal": 33,
- "timestamp": "2026-05-13T21:01:15.625Z",
- "id": "toolu_01FsZYoJ6ddwjVLWKMwYoyU6",
- "kind": "structured",
- "step_count": 7,
- "steps": [
- {
- "action": "tap",
- "elementRef": "e130",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e133",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e137",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e140",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e142",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e145",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e148",
- "postDelay": 0.25
- }
- ],
- "result": {
- "line": 112,
- "timestamp": "2026-05-13T21:01:20.872Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null,
- "structured": null,
- "content": "{\"schema\":\"xcodebuildmcp.output.ui-action-result\",\"schemaVersion\":\"2\",\"didError\":false,\"error\":null,\"data\":{\"summary\":{\"status\":\"SUCCEEDED\"},\"action\":{\"type\":\"batch\",\"stepCount\":7},\"artifacts\":{\"simulatorId\":\"5213C8D8-61D0-4CD7-B468-B463C6206C7D\"}}}"
- }
- }
- ],
- "raw_string_batch_attempts": [],
- "errors": [],
- "settings_window_calls": [
- {
- "ordinal": 20,
- "timestamp": "2026-05-13T21:00:04.466Z",
- "name": "mcp__xcodebuildmcp-dev__swipe",
- "input": {
- "withinElementRef": "e7",
- "direction": "up",
- "distance": 800
- },
- "result": {
- "timestamp": "2026-05-13T21:00:06.036Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 21,
- "timestamp": "2026-05-13T21:00:08.085Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:00:08.545Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 22,
- "timestamp": "2026-05-13T21:00:11.517Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:13.199Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 23,
- "timestamp": "2026-05-13T21:00:20.754Z",
- "name": "TodoWrite",
- "input": {
- "todos": [
- {
- "content": "Build and launch Weather app with mock data",
- "status": "completed",
- "activeForm": "Building and launching Weather app with mock data"
- },
- {
- "content": "Scroll to bottom of main screen",
- "status": "completed",
- "activeForm": "Scrolling to bottom of main screen"
- },
- {
- "content": "Tap settings button and reveal all options",
- "status": "in_progress",
- "activeForm": "Opening settings sheet"
- },
- {
- "content": "Invert all settings toggles",
- "status": "pending",
- "activeForm": "Inverting settings toggles"
- },
- {
- "content": "Close settings sheet",
- "status": "pending",
- "activeForm": "Closing settings sheet"
- },
- {
- "content": "Scroll to precipitation widget and tap",
- "status": "pending",
- "activeForm": "Tapping precipitation widget"
- },
- {
- "content": "Close precipitation widget",
- "status": "pending",
- "activeForm": "Closing precipitation widget"
- },
- {
- "content": "Open Location screen",
- "status": "pending",
- "activeForm": "Opening Location screen"
- },
- {
- "content": "Scroll location sheet to reveal all",
- "status": "pending",
- "activeForm": "Revealing all locations"
- },
- {
- "content": "Tap Portland location",
- "status": "pending",
- "activeForm": "Tapping Portland"
- },
- {
- "content": "Tap locations button again",
- "status": "pending",
- "activeForm": "Reopening locations"
- },
- {
- "content": "Search for a location",
- "status": "pending",
- "activeForm": "Searching for a location"
- },
- {
- "content": "Press hardware home button",
- "status": "pending",
- "activeForm": "Pressing home button"
- },
- {
- "content": "Open Safari",
- "status": "pending",
- "activeForm": "Opening Safari"
- },
- {
- "content": "Search for BBC.co.uk",
- "status": "pending",
- "activeForm": "Searching BBC.co.uk"
- },
- {
- "content": "Go to sport page",
- "status": "pending",
- "activeForm": "Navigating to sport page"
- },
- {
- "content": "Go to Premier League page",
- "status": "pending",
- "activeForm": "Navigating to Premier League"
- },
- {
- "content": "Go to tables",
- "status": "pending",
- "activeForm": "Going to tables"
- },
- {
- "content": "Click Brighton",
- "status": "pending",
- "activeForm": "Clicking Brighton"
- }
- ]
- },
- "result": {
- "timestamp": "2026-05-13T21:00:20.762Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 24,
- "timestamp": "2026-05-13T21:00:23.387Z",
- "name": "mcp__xcodebuildmcp-dev__tap",
- "input": {
- "elementRef": "e113"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:24.176Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 25,
- "timestamp": "2026-05-13T21:00:26.779Z",
- "name": "mcp__xcodebuildmcp-dev__wait_for_ui",
- "input": {
- "predicate": "settled",
- "settledDurationMs": 500,
- "timeoutMs": 5000
- },
- "result": {
- "timestamp": "2026-05-13T21:00:28.153Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 26,
- "timestamp": "2026-05-13T21:00:33.949Z",
- "name": "mcp__xcodebuildmcp-dev__swipe",
- "input": {
- "withinElementRef": "e123",
- "direction": "up",
- "distance": 400
- },
- "result": {
- "timestamp": "2026-05-13T21:00:35.629Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 27,
- "timestamp": "2026-05-13T21:00:37.754Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:00:38.303Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 28,
- "timestamp": "2026-05-13T21:00:42.712Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:43.315Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 29,
- "timestamp": "2026-05-13T21:00:49.819Z",
- "name": "mcp__xcodebuildmcp-dev__swipe",
- "input": {
- "withinElementRef": "e115",
- "direction": "up",
- "distance": 500
- },
- "result": {
- "timestamp": "2026-05-13T21:00:51.550Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 30,
- "timestamp": "2026-05-13T21:00:54.206Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:00:54.821Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 31,
- "timestamp": "2026-05-13T21:01:01.034Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:01:01.575Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 32,
- "timestamp": "2026-05-13T21:01:11.819Z",
- "name": "TodoWrite",
- "input": {
- "todos": [
- {
- "content": "Build and launch Weather app with mock data",
- "status": "completed",
- "activeForm": "Building and launching Weather app with mock data"
- },
- {
- "content": "Scroll to bottom of main screen",
- "status": "completed",
- "activeForm": "Scrolling to bottom of main screen"
- },
- {
- "content": "Tap settings button and reveal all options",
- "status": "completed",
- "activeForm": "Opening settings sheet"
- },
- {
- "content": "Invert all settings toggles",
- "status": "in_progress",
- "activeForm": "Inverting settings toggles"
- },
- {
- "content": "Close settings sheet",
- "status": "pending",
- "activeForm": "Closing settings sheet"
- },
- {
- "content": "Scroll to precipitation widget and tap",
- "status": "pending",
- "activeForm": "Tapping precipitation widget"
- },
- {
- "content": "Close precipitation widget",
- "status": "pending",
- "activeForm": "Closing precipitation widget"
- },
- {
- "content": "Open Location screen",
- "status": "pending",
- "activeForm": "Opening Location screen"
- },
- {
- "content": "Scroll location sheet to reveal all",
- "status": "pending",
- "activeForm": "Revealing all locations"
- },
- {
- "content": "Tap Portland location",
- "status": "pending",
- "activeForm": "Tapping Portland"
- },
- {
- "content": "Tap locations button again",
- "status": "pending",
- "activeForm": "Reopening locations"
- },
- {
- "content": "Search for a location",
- "status": "pending",
- "activeForm": "Searching for a location"
- },
- {
- "content": "Press hardware home button",
- "status": "pending",
- "activeForm": "Pressing home button"
- },
- {
- "content": "Open Safari",
- "status": "pending",
- "activeForm": "Opening Safari"
- },
- {
- "content": "Search for BBC.co.uk",
- "status": "pending",
- "activeForm": "Searching BBC.co.uk"
- },
- {
- "content": "Go to sport page",
- "status": "pending",
- "activeForm": "Navigating to sport page"
- },
- {
- "content": "Go to Premier League page",
- "status": "pending",
- "activeForm": "Navigating to Premier League"
- },
- {
- "content": "Go to tables",
- "status": "pending",
- "activeForm": "Going to tables"
- },
- {
- "content": "Click Brighton",
- "status": "pending",
- "activeForm": "Clicking Brighton"
- }
- ]
- },
- "result": {
- "timestamp": "2026-05-13T21:01:11.822Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 33,
- "timestamp": "2026-05-13T21:01:15.625Z",
- "name": "mcp__xcodebuildmcp-dev__batch",
- "input": {
- "steps": [
- {
- "action": "tap",
- "elementRef": "e130",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e133",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e137",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e140",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e142",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e145",
- "postDelay": 0.25
- },
- {
- "action": "tap",
- "elementRef": "e148",
- "postDelay": 0.25
- }
- ]
- },
- "result": {
- "timestamp": "2026-05-13T21:01:20.872Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 34,
- "timestamp": "2026-05-13T21:01:23.163Z",
- "name": "mcp__xcodebuildmcp-dev__screenshot",
- "input": {
- "returnFormat": "base64"
- },
- "result": {
- "timestamp": "2026-05-13T21:01:23.739Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 35,
- "timestamp": "2026-05-13T21:01:27.506Z",
- "name": "mcp__xcodebuildmcp-dev__snapshot_ui",
- "input": {},
- "result": {
- "timestamp": "2026-05-13T21:01:28.031Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 36,
- "timestamp": "2026-05-13T21:01:31.879Z",
- "name": "mcp__xcodebuildmcp-dev__tap",
- "input": {
- "elementRef": "e126"
- },
- "result": {
- "timestamp": "2026-05-13T21:01:32.759Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 37,
- "timestamp": "2026-05-13T21:01:35.902Z",
- "name": "mcp__xcodebuildmcp-dev__wait_for_ui",
- "input": {
- "predicate": "gone",
- "text": "Settings",
- "timeoutMs": 3000
- },
- "result": {
- "timestamp": "2026-05-13T21:01:39.013Z",
- "status": null,
- "error_code": null,
- "seq": null,
- "capture_count": null
- }
- },
- {
- "ordinal": 38,
- "timestamp": "2026-05-13T21:01:49.208Z",
- "name": "TodoWrite",
- "input": {
- "todos": [
- {
- "content": "Build and launch Weather app with mock data",
- "status": "completed",
- "activeForm": "Building and launching Weather app with mock data"
- },
- {
- "content": "Scroll to bottom of main screen",
- "status": "completed",
- "activeForm": "Scrolling to bottom of main screen"
- },
- {
- "content": "Tap settings button and reveal all options",
- "status": "completed",
- "activeForm": "Opening settings sheet"
- },
- {
- "content": "Invert all settings toggles",
- "status": "completed",
- "activeForm": "Inverting settings toggles"
- },
- {
- "content": "Close settings sheet",
- "status": "completed",
- "activeForm": "Closing settings sheet"
- },
- {
- "content": "Scroll to precipitation widget and tap",
- "status": "in_progress",
- "activeForm": "Tapping precipitation widget"
- },
- {
- "content": "Close precipitation widget",
- "status": "pending",
- "activeForm": "Closing precipitation widget"
- },
- {
- "content": "Open Location screen",
- "status": "pending",
- "activeForm": "Opening Location screen"
- },
- {
- "content": "Scroll location sheet to reveal all",
- "status": "pending",
- "activeForm": "Revealing all locations"
- },
- {
- "content": "Tap Portland location",
- "status": "pending",
- "activeForm": "Tapping Portland"
- },
- {
- "content": "Tap locations button again",
- "status": "pending",
- "activeForm": "Reopening locations"
- },
- {
- "content": "Search for a location",
- "status": "pending",
- "activeForm": "Searching for a location"
- },
- {
- "content": "Press hardware home button",
- "status": "pending",
- "activeForm": "Pressing home button"
- },
- {
- "content": "Open Safari",
- "status": "pending",
- "activeForm": "Opening Safari"
- },
- {
- "content": "Search for BBC.co.uk",
- "status": "pending",
- "activeForm": "Searching BBC.co.uk"
- },
- {
- "content": "Go to sport page",
- "status": "pending",
- "activeForm": "Navigating to sport page"
- },
- {
- "content": "Go to Premier League page",
- "status": "pending",
- "activeForm": "Navigating to Premier League"
- },
... diff truncated: showing 800 of 12059 linesYou can send follow-ups to the cloud agent here.
Add batch execution, wait predicates, runtime snapshot refs, and screen-hash unchanged responses so agents can drive AXe with fewer process launches and less repeated snapshot output. Tighten action validation, stale-snapshot recovery, compact rendering, and fixture coverage so UI automation flows are easier for agents to use reliably. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Return compact runtime snapshot captures after UI actions and derive next-step guidance from foreground runtime elements. This reduces reliance on full snapshots and screenshot verification loops while preserving structured output contracts. Co-Authored-By: Codex <noreply@openai.com>
Add coverage for runtime captures, post-action snapshot behavior, compact next-step rendering, and session-aware structured output handling. Co-Authored-By: Codex <noreply@openai.com>
Record the runtime snapshot and next-step guidance improvements for the upcoming release. Co-Authored-By: Codex <noreply@openai.com>
cameroncooke
force-pushed
the
cam/feat/ui-automation-runtime-parity
branch
from
c0a5d33 to
15edeec
Compare
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: SIMULATOR_UUID placeholders leak in wait_for_ui manifest
- Removed static nextSteps block from wait_for_ui.yaml to eliminate SIMULATOR_UUID placeholder leak, matching the fix already applied to snapshot_ui.yaml.
Or push these changes by commenting:
@cursor push 46bb7686f9
Preview (46bb7686f9)
diff --git a/manifests/tools/wait_for_ui.yaml b/manifests/tools/wait_for_ui.yaml
--- a/manifests/tools/wait_for_ui.yaml
+++ b/manifests/tools/wait_for_ui.yaml
@@ -9,18 +9,6 @@
version: '2'
routing:
stateful: true
-nextSteps:
- - label: Refresh runtime snapshot
- toolId: snapshot_ui
- params:
- simulatorId: SIMULATOR_UUID
- when: success
- - label: Wait again
- toolId: wait_for_ui
- params:
- simulatorId: SIMULATOR_UUID
- predicate: settled
- when: success
annotations:
title: Wait for UI
readOnlyHint: trueYou can send follow-ups to the cloud agent here.
Allow existing coordinate-based UI action payloads and UI hierarchy captures in the v2 schemas. Avoid publishing wait-for-ui next-step params when the wait fails, and remove an unused simulator test import. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Schema relaxes video FPS from positive integer to any number
- Restored integer type constraint and minimum: 1 validation to the fps field in videoRecordingCapture schema definition.
Or push these changes by commenting:
@cursor push 5e661b4638
Preview (5e661b4638)
diff --git a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
--- a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
+++ b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
@@ -183,7 +183,7 @@
"properties": {
"type": { "const": "video-recording" },
"state": { "enum": ["started", "stopped"] },
- "fps": { "type": "number" },
+ "fps": { "type": "integer", "minimum": 1 },
"outputFile": { "type": "string" },
"sessionId": { "type": "string" }
},You can send follow-ups to the cloud agent here.
Remove the wait_for_ui manifest nextSteps block so the tool relies on dynamic next-step params instead of leaking SIMULATOR_UUID placeholders. Co-Authored-By: Codex <noreply@openai.com>
Restore the capture-result schema fps constraint to a positive integer while keeping the UI hierarchy capture additions intact. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Redundant
stylevariable duplicates existingoutputStylecheck- Removed redundant
stylevariable and replaced usage withoutputStyle !== 'minimal'at line 412.
- Removed redundant
- ✅ Fixed: NaN silently passes through number array coercion for trailing commas
- Modified
coerceNumberArrayto filter out empty strings instead of converting them to NaN, preventing silent NaN propagation.
- Modified
Or push these changes by commenting:
@cursor push 52baba93d8
Preview (52baba93d8)
diff --git a/src/cli/__tests__/schema-to-yargs.test.ts b/src/cli/__tests__/schema-to-yargs.test.ts
--- a/src/cli/__tests__/schema-to-yargs.test.ts
+++ b/src/cli/__tests__/schema-to-yargs.test.ts
@@ -37,6 +37,6 @@
expect(coerce?.('23,18,14')).toEqual([23, 18, 14]);
expect(coerce?.('23, 18, 14')).toEqual([23, 18, 14]);
expect(coerce?.(['23', '18,14'])).toEqual([23, 18, 14]);
- expect(coerce?.('23,')).toEqual([23, Number.NaN]);
+ expect(coerce?.('23,')).toEqual([23]);
});
});
diff --git a/src/cli/register-tool-commands.ts b/src/cli/register-tool-commands.ts
--- a/src/cli/register-tool-commands.ts
+++ b/src/cli/register-tool-commands.ts
@@ -298,7 +298,6 @@
const outputStyle: OutputStyle = argv.style === 'minimal' ? 'minimal' : 'normal';
const socketPath = argv.socket as string;
const logLevel = argv['log-level'] as string | undefined;
- const style = argv.style as string | undefined;
const filePathRenderStyle = argv.filePathRenderStyle as FilePathRenderStyle | undefined;
const verboseOutput = argv.verbose === true;
@@ -409,7 +408,7 @@
runtime: 'cli',
outputStyle,
filePathRenderStyle,
- includeNextSteps: style !== 'minimal',
+ includeNextSteps: outputStyle !== 'minimal',
});
const writeJsonlFragment =
outputFormat === 'jsonl'
diff --git a/src/cli/schema-to-yargs.ts b/src/cli/schema-to-yargs.ts
--- a/src/cli/schema-to-yargs.ts
+++ b/src/cli/schema-to-yargs.ts
@@ -13,7 +13,8 @@
String(entry)
.split(',')
.map((item) => item.trim())
- .map((item) => (item === '' ? Number.NaN : Number(item))),
+ .filter((item) => item !== '')
+ .map((item) => Number(item)),
);
}You can send follow-ups to the cloud agent here.
Remove the redundant argv.style alias and use the normalized outputStyle value when deciding whether to include CLI next steps. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Duplicated schema definitions across two JSON schema files
- Extracted 7 shared type definitions to common.schema.json and updated both schemas to reference them, eliminating duplication.
Or push these changes by commenting:
@cursor push f0e0e2782a
Preview (f0e0e2782a)
diff --git a/schemas/structured-output/_defs/common.schema.json b/schemas/structured-output/_defs/common.schema.json
--- a/schemas/structured-output/_defs/common.schema.json
+++ b/schemas/structured-output/_defs/common.schema.json
@@ -564,6 +564,124 @@
"minLength": 1
},
"minItems": 1
+ },
+ "frame": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "x": { "type": "number" },
+ "y": { "type": "number" },
+ "width": { "type": "number" },
+ "height": { "type": "number" }
+ },
+ "required": ["x", "y", "width", "height"]
+ },
+ "runtimeActionName": {
+ "enum": ["tap", "typeText", "longPress", "touch", "swipeWithin"]
+ },
+ "runtimeElementRole": {
+ "enum": [
+ "application",
+ "button",
+ "cell",
+ "image",
+ "keyboard-key",
+ "list",
+ "menu",
+ "other",
+ "scroll-view",
+ "slider",
+ "switch",
+ "tab",
+ "text",
+ "text-field",
+ "window"
+ ]
+ },
+ "runtimeElementState": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "enabled": { "type": "boolean" },
+ "focused": { "type": "boolean" },
+ "selected": { "type": "boolean" },
+ "visible": { "type": "boolean" }
+ }
+ },
+ "runtimeElement": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "ref": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
+ "role": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElementRole" },
+ "label": { "type": "string" },
+ "value": { "type": "string" },
+ "identifier": { "type": "string" },
+ "frame": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/frame" },
+ "state": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElementState" },
+ "actions": {
+ "type": "array",
+ "items": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeActionName" }
+ }
+ },
+ "required": ["ref", "frame", "actions"]
+ },
+ "compactRuntimeSnapshot": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "type": { "const": "runtime-snapshot" },
+ "rs": { "const": "1" },
+ "screenHash": { "type": "string", "minLength": 1 },
+ "seq": { "type": "integer", "minimum": 0 },
+ "count": { "type": "integer", "minimum": 0 },
+ "targets": {
+ "type": "array",
+ "items": { "type": "string" }
+ },
+ "scroll": {
+ "type": "array",
+ "items": { "type": "string" }
+ },
+ "text": {
+ "type": "array",
+ "items": { "type": "string" }
+ },
+ "udid": { "type": "string" }
+ },
+ "required": ["type", "rs", "screenHash", "seq", "count", "targets", "scroll", "udid"]
+ },
+ "recoverableUiError": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "code": {
+ "enum": [
+ "SNAPSHOT_MISSING",
+ "SNAPSHOT_EXPIRED",
+ "SNAPSHOT_PARSE_FAILED",
+ "ELEMENT_REF_NOT_FOUND",
+ "TARGET_NOT_FOUND",
+ "TARGET_AMBIGUOUS",
+ "TARGET_NOT_ACTIONABLE",
+ "WAIT_TIMEOUT",
+ "UI_STATE_CHANGED",
+ "ACTION_FAILED"
+ ]
+ },
+ "message": { "type": "string" },
+ "recoveryHint": { "type": "string" },
+ "elementRef": { "type": "string" },
+ "candidates": {
+ "type": "array",
+ "items": {
+ "oneOf": [{ "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElement" }, { "type": "string" }]
+ }
+ },
+ "snapshotAgeMs": { "type": "integer", "minimum": 0 },
+ "timeoutMs": { "type": "integer", "minimum": 0 }
+ },
+ "required": ["code", "message", "recoveryHint"]
}
}
}
diff --git a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
--- a/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
+++ b/schemas/structured-output/xcodebuildmcp.output.capture-result/2.schema.json
@@ -9,17 +9,6 @@
}
],
"$defs": {
- "frame": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "x": { "type": "number" },
- "y": { "type": "number" },
- "width": { "type": "number" },
- "height": { "type": "number" }
- },
- "required": ["x", "y", "width", "height"]
- },
"uiHierarchyNode": {
"type": "object"
},
@@ -35,61 +24,11 @@
},
"required": ["type", "uiHierarchy"]
},
- "runtimeActionName": {
- "enum": ["tap", "typeText", "longPress", "touch", "swipeWithin"]
- },
- "runtimeElementRole": {
- "enum": [
- "application",
- "button",
- "cell",
- "image",
- "keyboard-key",
- "list",
- "menu",
- "other",
- "scroll-view",
- "slider",
- "switch",
- "tab",
- "text",
- "text-field",
- "window"
- ]
- },
- "runtimeElementState": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "enabled": { "type": "boolean" },
- "focused": { "type": "boolean" },
- "selected": { "type": "boolean" },
- "visible": { "type": "boolean" }
- }
- },
- "runtimeElement": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "ref": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
- "role": { "$ref": "#/$defs/runtimeElementRole" },
- "label": { "type": "string" },
- "value": { "type": "string" },
- "identifier": { "type": "string" },
- "frame": { "$ref": "#/$defs/frame" },
- "state": { "$ref": "#/$defs/runtimeElementState" },
- "actions": {
- "type": "array",
- "items": { "$ref": "#/$defs/runtimeActionName" }
- }
- },
- "required": ["ref", "frame", "actions"]
- },
"runtimeActionHint": {
"type": "object",
"additionalProperties": false,
"properties": {
- "action": { "$ref": "#/$defs/runtimeActionName" },
+ "action": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeActionName" },
"elementRef": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
"label": { "type": "string" }
},
@@ -108,7 +47,7 @@
"expiresAtMs": { "type": "integer", "minimum": 0 },
"elements": {
"type": "array",
- "items": { "$ref": "#/$defs/runtimeElement" }
+ "items": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElement" }
},
"actions": {
"type": "array",
@@ -127,31 +66,6 @@
"actions"
]
},
- "compactRuntimeSnapshot": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "type": { "const": "runtime-snapshot" },
- "rs": { "const": "1" },
- "screenHash": { "type": "string", "minLength": 1 },
- "seq": { "type": "integer", "minimum": 0 },
- "count": { "type": "integer", "minimum": 0 },
- "targets": {
- "type": "array",
- "items": { "type": "string" }
- },
- "scroll": {
- "type": "array",
- "items": { "type": "string" }
- },
- "text": {
- "type": "array",
- "items": { "type": "string" }
- },
- "udid": { "type": "string" }
- },
- "required": ["type", "rs", "screenHash", "seq", "count", "targets", "scroll", "udid"]
- },
"runtimeSnapshotUnchanged": {
"type": "object",
"additionalProperties": false,
@@ -200,43 +114,11 @@
"matches": {
"type": "array",
"items": {
- "oneOf": [{ "$ref": "#/$defs/runtimeElement" }, { "type": "string" }]
+ "oneOf": [{ "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/runtimeElement" }, { "type": "string" }]
}
}
},
"required": ["predicate", "matches"]
- },
- "recoverableUiError": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "code": {
- "enum": [
- "SNAPSHOT_MISSING",
- "SNAPSHOT_EXPIRED",
- "SNAPSHOT_PARSE_FAILED",
- "ELEMENT_REF_NOT_FOUND",
- "TARGET_NOT_FOUND",
- "TARGET_AMBIGUOUS",
- "TARGET_NOT_ACTIONABLE",
- "WAIT_TIMEOUT",
- "UI_STATE_CHANGED",
- "ACTION_FAILED"
- ]
- },
- "message": { "type": "string" },
- "recoveryHint": { "type": "string" },
- "elementRef": { "type": "string" },
- "candidates": {
- "type": "array",
- "items": {
- "oneOf": [{ "$ref": "#/$defs/runtimeElement" }, { "type": "string" }]
- }
- },
- "snapshotAgeMs": { "type": "integer", "minimum": 0 },
- "timeoutMs": { "type": "integer", "minimum": 0 }
- },
- "required": ["code", "message", "recoveryHint"]
}
},
"properties": {
@@ -274,7 +156,7 @@
},
{ "$ref": "#/$defs/uiHierarchyCapture" },
{ "$ref": "#/$defs/runtimeSnapshot" },
- { "$ref": "#/$defs/compactRuntimeSnapshot" },
+ { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/compactRuntimeSnapshot" },
{ "$ref": "#/$defs/videoRecordingCapture" },
{ "$ref": "#/$defs/runtimeSnapshotUnchanged" },
{ "$ref": "#/$defs/compactRuntimeSnapshotUnchanged" }
@@ -283,7 +165,7 @@
"diagnostics": {
"$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/basicDiagnostics"
},
- "uiError": { "$ref": "#/$defs/recoverableUiError" },
+ "uiError": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/recoverableUiError" },
"waitMatch": { "$ref": "#/$defs/waitMatch" }
},
"required": ["summary", "artifacts"]
diff --git a/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json b/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json
--- a/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json
+++ b/schemas/structured-output/xcodebuildmcp.output.ui-action-result/2.schema.json
@@ -9,92 +9,6 @@
}
],
"$defs": {
- "frame": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "x": { "type": "number" },
- "y": { "type": "number" },
- "width": { "type": "number" },
- "height": { "type": "number" }
- },
- "required": ["x", "y", "width", "height"]
- },
- "runtimeActionName": {
- "enum": ["tap", "typeText", "longPress", "touch", "swipeWithin"]
- },
- "runtimeElementRole": {
- "enum": [
- "application",
- "button",
- "cell",
- "image",
- "keyboard-key",
- "list",
- "menu",
- "other",
- "scroll-view",
- "slider",
- "switch",
- "tab",
- "text",
- "text-field",
- "window"
- ]
- },
- "runtimeElementState": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "enabled": { "type": "boolean" },
- "focused": { "type": "boolean" },
- "selected": { "type": "boolean" },
- "visible": { "type": "boolean" }
- }
- },
- "runtimeElement": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "ref": { "type": "string", "pattern": "^e[1-9][0-9]*$" },
- "role": { "$ref": "#/$defs/runtimeElementRole" },
- "label": { "type": "string" },
- "value": { "type": "string" },
- "identifier": { "type": "string" },
- "frame": { "$ref": "#/$defs/frame" },
- "state": { "$ref": "#/$defs/runtimeElementState" },
- "actions": {
- "type": "array",
- "items": { "$ref": "#/$defs/runtimeActionName" }
- }
- },
- "required": ["ref", "frame", "actions"]
- },
- "compactRuntimeSnapshot": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "type": { "const": "runtime-snapshot" },
- "rs": { "const": "1" },
- "screenHash": { "type": "string", "minLength": 1 },
- "seq": { "type": "integer", "minimum": 0 },
- "count": { "type": "integer", "minimum": 0 },
- "targets": {
- "type": "array",
- "items": { "type": "string" }
- },
- "scroll": {
- "type": "array",
- "items": { "type": "string" }
- },
- "text": {
- "type": "array",
- "items": { "type": "string" }
- },
- "udid": { "type": "string" }
- },
- "required": ["type", "rs", "screenHash", "seq", "count", "targets", "scroll", "udid"]
- },
"point": {
"type": "object",
"additionalProperties": false,
@@ -106,38 +20,6 @@
},
"direction": {
"enum": ["up", "down", "left", "right"]
- },
- "recoverableUiError": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "code": {
- "enum": [
- "SNAPSHOT_MISSING",
- "SNAPSHOT_EXPIRED",
- "SNAPSHOT_PARSE_FAILED",
- "ELEMENT_REF_NOT_FOUND",
- "TARGET_NOT_FOUND",
- "TARGET_AMBIGUOUS",
- "TARGET_NOT_ACTIONABLE",
- "WAIT_TIMEOUT",
- "UI_STATE_CHANGED",
- "ACTION_FAILED"
- ]
- },
- "message": { "type": "string" },
- "recoveryHint": { "type": "string" },
- "elementRef": { "type": "string" },
- "candidates": {
- "type": "array",
- "items": {
- "oneOf": [{ "$ref": "#/$defs/runtimeElement" }, { "type": "string" }]
- }
- },
- "snapshotAgeMs": { "type": "integer", "minimum": 0 },
- "timeoutMs": { "type": "integer", "minimum": 0 }
- },
- "required": ["code", "message", "recoveryHint"]
}
},
"properties": {
@@ -333,8 +215,8 @@
"diagnostics": {
"$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/basicDiagnostics"
},
- "capture": { "$ref": "#/$defs/compactRuntimeSnapshot" },
- "uiError": { "$ref": "#/$defs/recoverableUiError" }
+ "capture": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/compactRuntimeSnapshot" },
+ "uiError": { "$ref": "https://xcodebuildmcp.com/schemas/structured-output/_defs/common.schema.json#/$defs/recoverableUiError" }
},
"required": ["summary", "action", "artifacts"]
},You can send follow-ups to the cloud agent here.
There was a problem hiding this comment.
key_press and key_sequence missing routing.stateful: true, bypassing daemon in CLI mode
key_press.yaml and key_sequence.yaml omit routing: stateful: true, so CLI invocations take the direct path instead of routing through the daemon — inconsistent with every other UI-interaction tool in this PR (tap, touch, long_press, swipe, type_text, batch, snapshot_ui, wait_for_ui) which all declare routing.stateful: true.
Verification
Ran grep for routing: across all manifests/tools/*.yaml. key_press.yaml and key_sequence.yaml have no routing block. All other stateful UI automation tools introduced or modified in this PR (batch, tap, long_press, swipe, touch, type_text, snapshot_ui, wait_for_ui) include routing:\n stateful: true. In tool-invoker.ts executeTool(), the daemon branch is entered only when opts.runtime === 'cli' && tool.stateful. Without the manifest flag, tool.stateful is falsy and the CLI falls through to the direct invocation path, silently bypassing the daemon for HID key operations that require a live simulator session — violating the 'Stateful CLI tools route through daemon only when routing.stateful requires it' guardrail.
Identified by Warden xcodebuildmcp-runtime-boundary-review
Document that CLI JSON simulator-name resolution failures now use the structured error envelope instead of plain stderr. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Keep simulator launch failures on the simulator launch path even when simulator-name resolution fails before a simulator ID exists. This prevents text and structured output from mislabeling those failures as macOS launches. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Remove stale coordinate-first accessibility hierarchy tips so agents are steered toward runtime snapshot refs instead of raw frame math. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Return a fresh runtime snapshot after successful mutating UI automation actions instead of preserving cached captures for selected paths. This keeps refs and visible state consistent after taps, key input, button presses, and batch actions. Update UI automation snapshot fixtures, normalize volatile SpringBoard clock rows, and align smoke-test expectations with current schema versions and tap contracts. Co-Authored-By: Codex <noreply@openai.com>
Collaborator
Author
|
Follow-up on the non-inline runtime snapshot feedback: fixed in 6b98258. Successful mutating UI automation actions now attempt a fresh post-action runtime snapshot instead of preserving cached action snapshots. That covers the stale batch snapshot concern and the touch down/up missing-snapshot path; gesture already captured a fresh snapshot and remains covered. Validation included lint, typecheck, format check, build, npm test, the regenerated UI automation snapshot suite, smoke tests, and a manual Calculator UI automation smoke test. |
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Unused
verboseparameter never affects output behavior- Changed runtimeSnapshot from hardcoded 'compact' to conditional 'options.verbose ? "full" : "compact"' and updated tests to verify verbose mode produces full output.
Or push these changes by commenting:
@cursor push bc97f1a3b8
Preview (bc97f1a3b8)
diff --git a/src/cli/__tests__/register-tool-commands.test.ts b/src/cli/__tests__/register-tool-commands.test.ts
--- a/src/cli/__tests__/register-tool-commands.test.ts
+++ b/src/cli/__tests__/register-tool-commands.test.ts
@@ -1212,7 +1212,7 @@
expect(output.data.waitMatch.matches).toEqual(['e11|none|text|No matches||']);
});
- it('keeps UI action runtime snapshots compact for verbose JSON output', async () => {
+ it('renders full UI action runtime snapshots for verbose JSON output', async () => {
mockInvokeDirectThroughHandler();
const stdoutChunks = captureStdoutChunks();
@@ -1271,21 +1271,23 @@
const output = JSON.parse(stdoutChunks.join('')) as {
schema: string;
- data: { capture: { targets: string[]; elements?: unknown[] } };
+ data: { capture: { targets?: string[]; elements?: unknown[] } };
};
expect(output.schema).toBe('xcodebuildmcp.output.ui-action-result');
expect(output.data.capture).toEqual(
expect.objectContaining({
type: 'runtime-snapshot',
- rs: '1',
- targets: ['e2|tap|button|Continue||'],
- scroll: ['e1|swipe|application|Weather||'],
+ protocol: 'rs/1',
+ simulatorId: 'SIMULATOR-1',
+ screenHash: 'screen-hash',
+ seq: 1,
}),
);
- expect(output.data.capture.elements).toBeUndefined();
+ expect(output.data.capture.elements).toBeDefined();
+ expect(output.data.capture.elements).toHaveLength(2);
});
- it('keeps capture runtime snapshots compact for verbose JSON output', async () => {
+ it('renders full capture runtime snapshots for verbose JSON output', async () => {
mockInvokeDirectThroughHandler();
const stdoutChunks: string[] = [];
vi.spyOn(process.stdout, 'write').mockImplementation((chunk) => {
@@ -1337,17 +1339,20 @@
const output = JSON.parse(stdoutChunks.join('')) as {
schema: string;
- data: { capture: { scroll: string[]; elements?: unknown[] } };
+ data: { capture: { scroll?: string[]; elements?: unknown[] } };
};
expect(output.schema).toBe('xcodebuildmcp.output.capture-result');
expect(output.data.capture).toEqual(
expect.objectContaining({
type: 'runtime-snapshot',
- rs: '1',
- scroll: ['e1|swipe|application|Weather||'],
+ protocol: 'rs/1',
+ simulatorId: 'SIMULATOR-1',
+ screenHash: 'screen-hash',
+ seq: 1,
}),
);
- expect(output.data.capture.elements).toBeUndefined();
+ expect(output.data.capture.elements).toBeDefined();
+ expect(output.data.capture.elements).toHaveLength(1);
});
it('writes one NDJSON line per domain fragment for jsonl output and omits the final envelope', async () => {
diff --git a/src/cli/register-tool-commands.ts b/src/cli/register-tool-commands.ts
--- a/src/cli/register-tool-commands.ts
+++ b/src/cli/register-tool-commands.ts
@@ -111,7 +111,7 @@
nextSteps: session.getNextSteps?.(),
nextStepRuntime: session.getNextStepsRuntime?.(),
outputStyle: options.outputStyle,
- runtimeSnapshot: 'compact',
+ runtimeSnapshot: options.verbose ? 'full' : 'compact',
},
)
: toStructuredEnvelope(You can send follow-ups to the cloud agent here.
Allow drag to use refs that support either direct touch or swipeWithin so application and window scroll targets are not rejected before drag point selection can run. Keep swipe semantic-only while adding explicit migration guidance for legacy coordinate arguments, and update stale smoke coverage to use runtime refs instead of coordinates. Cache scroll descendant facts during runtime snapshot normalization so scroll inference preserves behavior without repeated full-tree scans. Co-Authored-By: Codex <noreply@openai.com>
Keep UI action schema v2 compact-only and add schema v3 for verbose JSON output that includes full runtime snapshots. This preserves the published v2 contract while making --verbose meaningful for UI actions. Co-Authored-By: Codex <noreply@openai.com>
Keep runtime element state emitted for every normalized element, but remove the misleading optional return path from readState. enabled and visible are intentional runtime facts used by snapshots and hashes. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Missing newline before function definition in shell script
- Added a blank line between the closing fi statement and the validate_axe_version_metadata function definition to improve script readability.
Or push these changes by commenting:
@cursor push 76090221f6
Preview (76090221f6)
diff --git a/scripts/bundle-axe.sh b/scripts/bundle-axe.sh
--- a/scripts/bundle-axe.sh
+++ b/scripts/bundle-axe.sh
@@ -324,6 +324,7 @@
echo "⚠️ Skipping AXe binary verification on non-macOS (detected $OS_NAME)"
AXE_VERSION="unknown (verification skipped)"
fi
+
validate_axe_version_metadata() {
if [ "$AXE_VERSION" = "unknown (verification skipped)" ]; then
returnYou can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 71d77fb. Configure here.
Add spacing before the AXe metadata validation function so the helper does not visually merge with the preceding platform conditional. Co-Authored-By: Codex <noreply@openai.com>
Use the runtime viewport resolver for drag coordinate clamping instead of assuming the first snapshot element is the viewport. This keeps directional drags bounded to the actual app/window frame when AXe returns another root element first. Co-Authored-By: Codex <codex@openai.com>
Use locale-insensitive lowercase normalization for runtime snapshot label filtering in both structured output and text rendering. This keeps internal target filtering deterministic across host locales. Co-Authored-By: Codex <codex@openai.com>
| | 'ELEMENT_REF_NOT_FOUND' | ||
| | 'TARGET_NOT_FOUND' | ||
| | 'TARGET_AMBIGUOUS' | ||
| | 'TARGET_NOT_ACTIONABLE' |
There was a problem hiding this comment.
candidates in UiAutomationRecoverableError is unbounded, unlike the capped snapshot targets list
When a TARGET_NOT_ACTIONABLE error is returned, candidates is populated with every element in the snapshot that supports the requested action — no limit is applied — while the main compact snapshot enforces COMPACT_RUNTIME_TARGET_LIMIT = 64. A complex UI can produce error payloads with hundreds of candidate entries.
Evidence
candidates?: RuntimeElementV1[]at line 111 carries no size annotation or bound.- In
snapshot-ui-state.ts, the field is set tosnapshot.payload.elements.filter(candidate => requiredActions.some(...))with no.slice(). - In
structured-output-envelope.tsline 354 the candidates are mapped throughcompactRuntimeElementCandidatebut there is no.slice()— compare withtoRuntimeSnapshotCompactCaptureat lines 239–240 which applies.slice(0, COMPACT_RUNTIME_TARGET_LIMIT)(64), and scroll/text lists which are similarly capped at 32/64. - For a UI whose accessibility tree contains 500+ tappable elements, a single
TARGET_NOT_ACTIONABLEerror would emit all 500 compacted rows into the tool output.
Identified by Warden find-bugs · UWP-NKX
Comment on lines
+799
to
+802
| const isMac = | ||
| !isSimulator && | ||
| !isDevice && | ||
| (!result.didError || result.error === 'Failed to launch macOS app.'); |
There was a problem hiding this comment.
Platform detection coupled to hardcoded error message string
Determining isMac based on result.error === 'Failed to launch macOS app.' tightly couples the renderer to the exact error string set in buildLaunchError; if that message ever changes, macOS error results will silently render as generic 'Launch App' failures with the wrong header and no params.
Evidence
buildLaunchErrorinapp-lifecycle-results.ts:109–111is the only producer oferrorvalues forlaunch-result; it sets'Failed to launch macOS app.'vs'Failed to launch app.'based onisMacLaunch(artifacts, options.target).isMacLaunchalready makes a clean structural check (!('simulatorId' in artifacts) && !('deviceId' in artifacts)), which is the same information already available to the renderer viaisSimulator/isDevice.- The renderer's new guard (
!result.didError || result.error === 'Failed to launch macOS app.') creates a second, redundant path to the same determination but based on a mutable string rather than the artifacts. - Any future change to the error message string in
buildLaunchErrorwould makeisMacevaluate tofalsefor macOS error cases, causing the title to fall back to'Launch App'and the params block to enter the simulator/device branch with nosimulatorIdto show.
Identified by Warden code-review · AM7-8VG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add rs/1 runtime UI automation parity for simulator workflows.
This PR changes UI automation responses from "the action finished" into reusable next-action context. UI action tools can now return compact runtime captures with stable element refs, screen hashes, action targets, scroll targets, and app-agnostic next-step guidance. The goal is to let agents continue from the previous tool result instead of repeatedly calling
snapshot_uior falling back to screenshots for routine verification.Why this matters: the previous flow made agents rediscover the same UI after nearly every action. That inflated token use, encouraged extra screenshots/snapshots, and made scroll/tap decisions depend too much on agent guesswork. Runtime captures keep the structured-output envelope intact while giving the model enough foreground UI context to choose the next control directly.
The main implementation changes are:
scrollrows rather than invented refs.key_pressandkey_sequencethrough the daemon path so keyboard interactions share the same simulator/session state behavior as the other UI automation tools.data.capture.This is intentionally app-agnostic. The business rules do not special-case the Weather sample app or any specific UI. They reason from runtime UI data: visible/actionable controls, AX roles, identifiers when available, tree depth, containment, foreground surfaces, and nearby control relationships.
The companion contributor docs PR is: getsentry/xcodebuildmcp.com#16
Manual Claude comparison against the deterministic Weather UI task showed the intended improvement:
Local validation covered formatting, linting, type checking, build, focused UI automation tests, schema fixture validation, focused CLI JSON snapshots, and the full Vitest suite.